fix(credits): meter LLM credits when a turn fails after a side-effect - #786
fix(credits): meter LLM credits when a turn fails after a side-effect#786sweetmantech wants to merge 1 commit into
Conversation
runAgentWorkflow charged the account + wrote the usage_events audit row
only on `if (result.responseMessage)` after runAgentStep returned. The
customer email is a `send_email` tool call executed INSIDE runAgentStep,
so a turn that emailed the customer and then threw (or returned no
responseMessage) hit the try's `finally` with no catch and skipped
billing entirely: email out, wallet not debited, no usage_events row.
Observed 2026-07-23: a scheduled briefing on moonshotai/kimi-k3 emailed
the customer at 20:54 but wrote 0 rows with a model_id (vs the sonnet-5
run's 1). Both runs' model-independent scrape/tool charges were
identical; only the LLM line went missing.
Fix: extract the charge into an idempotent `chargeTurn` and call it both
on the normal path and as a `finally` backstop. A turn that produced a
customer-facing side-effect now always lands a wallet debit + audit row;
ZERO_USAGE with no gateway cost floors to the 1c minimum, so a failed
turn still writes a model_id row. Normal + user-abort paths bill exactly
once (chargeTurn guards on `charged`); the original error still
propagates after the backstop runs.
Flips two tests that codified the gap ("does NOT bill when no
responseMessage" / "...when it throws") to assert the turn is now billed.
20 workflow tests + 129 workflows/credits domain tests pass; eslint clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reached
Next review available in: 6 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (1)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
1 issue found across 2 files
Confidence score: 3/5
- In
app/lib/workflows/runAgentWorkflow.ts, the workflow can mark a turn as completed before credit debit is confirmed, andhandleChatCreditscurrently swallows failures, so transient billing errors may leave side-effecting turns unbilled with no retry signal. HavehandleChatCreditsreturn/propagate a success result and only set the completion flag after a confirmed debit (or trigger a compensating/retry path).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="app/lib/workflows/runAgentWorkflow.ts">
<violation number="1" location="app/lib/workflows/runAgentWorkflow.ts:118">
P2: A transient credit-recording failure still leaves a side-effecting turn unbilled: this flag is set before the debit outcome is known, while `handleChatCredits` swallows failures. Return/propagate a success result and mark the turn charged only after a successful debit, so the finally path can retry a failed normal-path attempt.</violation>
</file>
Architecture diagram
sequenceDiagram
participant Workflow as runAgentWorkflow
participant AgentStep as runAgentStep
participant Billing as chargeTurn (NEW)
participant Credits as handleChatCredits
participant DB as Database
participant Email as send_email Tool
Note over Workflow,Email: Turn execution — agent may call tools (e.g. send_email) mid-step
Workflow->>AgentStep: Execute step with assistantMessageId
AgentStep->>Email: Customer-facing tool call (side-effect)
alt Normal path: AgentStep returns responseMessage
AgentStep-->>Workflow: result with responseMessage.metadata
Workflow->>Billing: chargeTurn() — idempotent
Billing->>Credits: handleChatCredits(usage=real)
Credits->>DB: deduct_credits_with_audit (wallet + usage_events)
DB-->>Credits: OK
Credits-->>Billing: done
Billing-->>Workflow: charged=true
Workflow->>Workflow: Auto-commit + push (if natural finish)
else AgentStep returns without responseMessage (no message, but email may have fired)
AgentStep-->>Workflow: result, responseMessage=undefined
Workflow->>Billing: chargeTurn() — idempotent
Billing->>Credits: handleChatCredits(usage=ZERO_USAGE → 1¢ floor)
Credits->>DB: deduct_credits_with_audit (model_id row, 1¢ min)
DB-->>Credits: OK
Credits-->>Billing: done
Billing-->>Workflow: charged=true
else AgentStep throws after email sent
AgentStep-->>Workflow: Error thrown
Workflow->>Billing: chargeTurn() — idempotent (in finally)
Billing->>Credits: handleChatCredits(usage=ZERO_USAGE → 1¢ floor)
Credits->>DB: deduct_credits_with_audit (model_id row, 1¢ min)
DB-->>Credits: OK
Credits-->>Billing: done
Billing-->>Workflow: charged=true
Workflow->>Workflow: Error propagates
end
Note over Workflow: Finally block — always runs
Workflow->>Billing: chargeTurn() — idempotent guard (already charged, no-op)
Billing-->>Workflow: skipped (charged=true)
Workflow->>Workflow: cleanup: clearChatActiveStream + commitChatTurn (parallel)
alt Error path (if AgentStep threw)
Workflow-->>Workflow: Error still propagates after billing + cleanup
end
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| let charged = false; | ||
| const chargeTurn = async () => { | ||
| if (charged) return; | ||
| charged = true; |
There was a problem hiding this comment.
P2: A transient credit-recording failure still leaves a side-effecting turn unbilled: this flag is set before the debit outcome is known, while handleChatCredits swallows failures. Return/propagate a success result and mark the turn charged only after a successful debit, so the finally path can retry a failed normal-path attempt.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At app/lib/workflows/runAgentWorkflow.ts, line 118:
<comment>A transient credit-recording failure still leaves a side-effecting turn unbilled: this flag is set before the debit outcome is known, while `handleChatCredits` swallows failures. Return/propagate a success result and mark the turn charged only after a successful debit, so the finally path can retry a failed normal-path attempt.</comment>
<file context>
@@ -97,35 +97,47 @@ export async function runAgentWorkflow(input: RunAgentWorkflowInput): Promise<vo
+ let charged = false;
+ const chargeTurn = async () => {
+ if (charged) return;
+ charged = true;
+ const metadata = result?.responseMessage?.metadata as AgentMessageMetadata | undefined;
+ await handleChatCredits({
</file context>
Verification — preview + CI + unitPreview: CI on Verified (unit, RED→GREEN)The two tests that codified the gap were flipped to assert the correct behavior, and fail on
Not verified by live preview curl — and why (being explicit)This is an internal change to the Observable acceptance (post-merge)The bug already reproduced in prod: Brauxelion's Kimi-K3 briefing (account |
1 participant
Tracked by chat#1885 (billing-integrity row; sibling to api#782/#783 — recommend splitting into a dedicated billing-integrity tracker).
Problem
runAgentWorkflowcharged the account + wrote theusage_eventsaudit row only atapp/lib/workflows/runAgentWorkflow.tsL119, gated onif (result.responseMessage)afterrunAgentStepreturned — inside atrywhose only companion is afinally(nocatch).The customer email is a
send_emailtool call executed insiderunAgentStep(mid-turn). So a turn that emails the customer and then throws — or returns without aresponseMessage— skips billing entirely: email out, wallet not debited, nousage_eventsrow.getCreditUsage/handleChatCreditscan't be the cause: an unknown/zero-priced model still writes a ≥1¢ row withmodel_id(Math.max(1, …)). Zero rows ⇒handleChatCreditswas never called.Evidence (2026-07-23)
A scheduled briefing for account
fade8fd2-…on the newmoonshotai/kimi-k3default emailed "Brauxelion Weekly" at 20:54 but wrote 0usage_eventsrows with amodel_id. The earlieranthropic/claude-sonnet-5run wrote 1 (106¢, 2.63M in / 31.8k out). Both runs' model-independent scrape/tool charges (235¢,model_id=null) were identical — only the LLM line vanished. With Kimi now the default for every headless briefing, this risks systemic email-but-not-bill with no audit trail.Fix
Extract the charge into an idempotent
chargeTurn()and call it (a) on the normal path and (b) as afinallybackstop. A turn that produced a customer-facing side-effect now always lands a wallet debit + audit row;ZERO_USAGEwith no gateway cost floors to the 1¢ minimum, so a failed turn still writes amodel_idrow.chargeTurnguards on achargedflag so the normal + user-abort paths bill exactly once. The original error still propagates after the backstop runs.Behavior
model_idrow) ✅send_emailfiredmodel_idrow) ✅; error still propagatesTests
TDD red→green. Flipped the two tests that codified the gap ("does NOT bill when no responseMessage" / "…when it throws") to assert the turn is now billed with the 1¢ floor.
pnpm exec vitest run app/lib/workflows/__tests__/runAgentWorkflow.test.ts→ 20 passedpnpm exec vitest run app/lib/workflows lib/credits→ 129 passedpnpm exec eslinton both changed files → cleanNo API contract change → no docs PR (a billing side-effect, like api#782). Note:
tsc --noEmithas 203 pre-existing errors onmain(trigger/tasks test fixtures); this PR adds none, andrunAgentWorkflow.tsis type-clean.Follow-up (out of scope)
Confirm why Kimi returned no
responseMessagein this workflow (Vercel workflow run logs:finishReason/ thrown error) — that's a model-integration question separate from this billing-integrity fix.🤖 Generated with Claude Code
Summary by cubic
Ensures LLM credits are charged when a turn fails or returns no response after a side-effect, preventing email-but-not-billed gaps. Billing now always records a
usage_eventsrow with a 1¢ floor.chargeTurn()and invoke it on success and infinally.model_idrow.Written for commit 007840d. Summary will update on new commits.